# Small-scale pretraining

This is a streamlined version based on Meta's Llama-3.2-1B model, specifically pretrained for Amharic with 400 million parameters and a context length of 1024 tokens.

A small CLIP model based on ViT architecture, suitable for zero-shot image classification tasks, trained on the LAION-400M dataset.

1.3B parameter GPT-NeoX model pretrained on 31.3GB Vietnamese data

RoBERTa series models pretrained on datasets of varying scales (1M-1B tokens), including BASE and MED-SMALL specifications

IT5 is the first family of sequence-to-sequence Transformer models pretrained at scale for Italian, following the approach of the original T5 model.

A RoBERTa base model pre-trained on 1B tokens with a validation perplexity of 3.93, suitable for English text processing tasks.

RoFormer is a Chinese Transformer model enhanced with Rotary Position Embedding, suitable for text infilling tasks.

This is a GPT2-small model pretrained on Indonesian Wikipedia data, specializing in Indonesian text generation tasks.

This is a RoBERTa model pretrained on Kinyarwanda datasets using Masked Language Modeling (MLM) objective, with case-insensitive tokenization during pretraining.

A RoBERTa model pretrained on a small-scale dataset of 1M tokens, using the MED-SMALL architecture, suitable for text understanding tasks.